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Abstract 

Despite its old age, Lisp remains mysterious to many of its admirers. 
The mysteries on one hand fascinate the language, on the other hand 
also obscure it. Following Stoyan but paying attention to what he has 
neglected or omitted, in this first essay of a series intended to unravel these 
mysteries, we trace the development of Lisp back to its origin, revealing 
how the language has evolved into its nowadays look and feel. The insights 
thus gained will not only enhance existent understanding of the language 
but also inspires further improvement of it. 


1 Introduction 


You have to know the past to 
understand the present. 

Carl Sagan 


Graham 2001 1 uncovered “the roots of Lisp” 0 and in particular, showed us “the 
surprise’il that a meta-circular interpreter for the language can be readily con¬ 
structed. This surprising result, which he nominated as “the defining quality of 
Lisp”, that the language “can be written in itself” [ibid, p. 1], has attracted a lot 
of language enthusiasts to Lisp. However, when they ask about the source of this 
surprise, they get answers such as that the language is Turing-complete or that 
programs are (manipulable as) data in the language. These answers, although 
succinct, are nebulous. To a large extent, the language remains mysterious. 

We intend in a series of essays to unravel the mysteries of Lisp. In the past 
decades, some scholars have tried to do so and achieved the goal to various 
degree. The most remarkable is probably Stoyan w ho carefully studi e d the 
history of Lisp for better understanding the language 19791 Il984 Il991 . 2007 . 


^Throughout this text, we will use the modern name Lisp for the language. However, when 
mentioning historical dialects or quoting text about them, we will stick to the ancient name 

LISP. _ _ 

^The one who first did so was of course [McCarth^ |l959l| . 
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2008l| . Holding the same position, we believe that to unravel the mysteries of 


Lisp, we must trace its development back to its origin. So this work could be 
considered a continuation of that by Stoyan. We believe most of what we discuss 
here and will discuss in the following series is folklore knowledge. Moreover, it 
must have already been investigated by Stoyan. However, we intend not to 
repeat what Stoyan has done, but to complement his work by gathering what 
he has neglected or omitted. Our main contribution is exhibiting our findings 
in one place and interpreting them in both historical and modern contexts. 

In this first essay of the series, we will look into the early development 
of Lisp. We will in particular lay out how the language has evolved into its 
nowadays look and feel. On just these aspects, a few pointers suffice to show 
that our grasp of the language is not thorough: we may have heard that Lisp 
became based on S-expressions more by accident, but have not considered any 
less-accidental factor; we may have learned that lists are constructed from pairs 
in Lisp, but not questioned the rationality of this actuality; we may have read 
that Lisp used to have a kind of expressions other than S-expressions for writing 
programs, but not investigated the context of their existence. 

As far as we know, these issues have not, if ever, been satisfactorily ad- 
dresse d. A probable re ason is that most of us never go beyond the landmark 
paper McCarthy 1960l| that systematically described the language. This pa- 

i )er, however, wa s not the first systematic description of Lisp. Its earlier draft 
McCarthy 1959l| . published as an AI memorandum, was. It is this AI memo 
on which we will focus our attention, and prior AI me mos in which we will seek 
useful clues. In our opinion, McCarthy pre sented in |l959l| a better-designed 
system than the later-determined version in [i 96 n| . which evidences again that 
the development of a system may not necessarily be an advancement but rather 
a regression. 


2 Toward S-expression Lisp 


In this section, we will look into the early development of Lisp, to a large extent 
in chronological order. We will focus our attention mainly on material related 
to the three issues we set out to address. For a mor e-general trea t ment of the 
early histor y of Lisp, the reade r is re ferred to either IStoya 3 |l979t nisi: I 1991 I 
2007t l2008l| or lMcCarthyl |l978l: ll 


2.1 An algebraic language 

Although according to IStoyanI |l984 1, the incubation of Lisp could date back to 


1956, it was in 1958 when the 1st AI memo was published that “an algebraic 
language for the m anipulation of symbolic expressions” (not named LISP yet) 
McCarth^ll958al| hatched out. McCarthy put in his new algebraic language 


most of the features he prop osed “for the Volume 2 (V2)” of the International 


Algebraic 


le leatures ne prop c 
Languag^ [l958cl |. 


Among these features, of particular interest to us 


®IAL, later known as ALGOL, the ALGOrithmic Language 
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is the proposal of an intermediate language that uniforms function position: a 
function-designating expression, even an operator, appears always in the head 
position, as in /(ci, ... .Cn). Another intriguing feature is representing both 
sequences and expressions as lists implemented using series of machine words on 
IBM 704. From this vivid description, we can already see a prototype of Lisp. 
However, note a few things. 

First, list was introduced as a kind of data structure, not yet abstracted as 
a data type. We diverge from Stoyan on thi s point. Stoyan held that “lists 
were not regarded as data structures” [1984 p. 304]. We believe they were. 


since “a number o f interesting and useful operations on lists have been defined” 
McCarth^ Il958al p. 5]. They were used for constructiiw lists and selecting 


components. The omission of lists “as a kind of quantity’^ was because “most 
of the calculations we actually perform cannot as yet be described in terms 
of these operations” and “it still seems to be necessary to compute with the 
addresses of the elements of the lists” [ibid, p. 5]. Although it is not clear what 
McCarthy meant by “cannot as yet be described” here, it is obvious that he felt 
that these operations were too low-level. 

Second, symbolic expressions were not part of the algebraic language. They 
belonged in both the intermediate language and the language of discourse (En¬ 
glish plus mathematics). McCarthy here c hanged for symbolic expressions 




1958cl| to the sequence notation 


from the function notation / (ei, .. 

(/,ei, ... ,e„), which resembled the prefix notation except explicit parenthe- 
sization. 

Third, the alg ebraic language its elf used mixfix notation. Altho ugh not ex¬ 
plicitly stated in McCarth^ Il958aj . according to the proposal in [McCarthvI 
ll958cJ | , programs in the algebraic language were supposed to be translated into 
the intermediate language and then further translated into the assembly lan¬ 
guage or machine language. Thus the first-stage translation will turn algebraic 
expressions in mixfix notation completely into symbolic expressions in prefix 
notation. 

The algebraic languag e received t wo revisions docu mented respectively in 
AIM-3 McCarthy I958d and AIM-4 McCarthy 1958bj |. AIM-3 made explicit 
that IBM-704 word sequences were internal representations of algebraic expres¬ 
sions while symbolic expressions external. Following the distinction was an 
attempt to formally define symbolic expressions, spontaneously called “external 
expressions” 0 

1 . A symbol is an [external] expression. 

2. If Cl, 62 , ..., e„ are [external] expressions, so is (ci , 62 , ... ,e„). 


McCarthy further noted the distinction between e and (e). But the special case 
for empty sequence was missing. Still, symbolic expressions were not admitted 
into the algebraic language. However, operations on lists were distilled into 

^In the context, the word ‘quantity’ probably meant datum or literal. Taking into account 
the absence of symbolic expressions from the algebraic language (discussed soon), this seems 
a plausible interpretation. 

^Our edits are put in square brackets. 
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more or less cons , car and cdr. In AIM-4, the name LISP, for “List Processor' 
McCarthvlll958bl . p. 9], first occurred. 


2.2 A list processor 

In th e 8th AI memo McCarth^ll959l | , the draft of the landmark paper McCarth^ 
Il 96 nj . symbolic expressions finally entered the algebraic language. Lisp began 
to feature two systems of notation at the source level: S-expressions (short for 
Symbolic expressions) and F- expression^ (for Functional expressions), which 
respectively correspond to the source-level forms of data and programs. 

In th is me mo, McCarthy presented a system different from what was later 
1963, _ even diffe rent from what we know today. The definition 


given m 


of S-expressions McCarth'\lll959l p. 3], now complete, is quoted below 0 


1 . The atomic symbols [...] are S-expressions. 

2. A null expression [()] is also admitted. 

3. If e is an S-expression[,] so is (e). 

4. If d and ([cs]) are S-expressions[,] so is (ei,[es]). 


Cs in the 4th clause might refer to a sequence of S-expressions “e 2 , ... ,e„” 
where n > 2. In that case, the rule gives us (ei,e 2 , ... ,e„). This notation 
for lists is almost the same as what we know today except that it used commas 
rather than merely spaces to separate list elements. 

According to this definition, valid compound S-expressions include only what 
we now call proper lists that always terminate with (), no ordered pairs, and 
naturally nor improper lists that do not end with 0 0 

As regards the notation for functional expressions, McCarthy switched to 
square brackets and semi- colons “since parentheses and commas have been pre¬ 
empted” by S-expressions 19591 p. 3]. This switch was a move away from the fa¬ 
miliar mathematical notation used since AIM-1, where f (ei, ... ,e„) were used 
for functional expressions, and (ei, ... ,e„) for symbolic expressions^ Had Mc¬ 
Carthy retained the notation in AIM-1 or reversed the notation in AIM-8 for S- 
expressions and F-expressiont0 — in other words, if he used square brackets and 
semi-colons for S-expressions while parentheses and commas for F-expressions, 


^‘F-exDressk)ry_[McCart^ ll95Sl . p. 13] was the original name of M-expressions (for Meta¬ 
expressions) iMcCarth^IlQ^ . p. 187]. We choose the old name for two reasons: (1) meta 
is relative, the name M-expression becomes misleading once F-expressions are taken to the 
level of object language; (2) the term F-expression clearly indicates that functional expres¬ 
sions describe functions of S-expressions, which McCarthy called S-functions [m, p. 1]. 
Nevertheless, in directly quoted text, we will keep the terms used in the source. 

^We have slightly edited the original definition to fit modern typographic style. In partic¬ 
ular, we use 0 in place of the hand-written A for the null expression. Again, our edits are put 
in square brackets. 

®From now on, when the word ‘list’ occurs without any qualifier, it means proper list, as 
has already been the case. 

^We could not see how this notation might cause any serious problem. 

^^Interestingly, a later Lisp dialect called M-LISP, which his inventor advertised as a “hy¬ 
brid of McCarthy’s original M- expression LISP an d Scheme”, did reverse the notation for 
S-expressions and F-expressions [Mullej[l99l] : • 
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that is, [ei ; ... ;e„] instead of (ei , ... ,e„), and f (ei , ... ,e„) rather than 
f [ei ; ... ; e„] — Lisp would have a more-mathematical flavor, which would in 
turn better justi fy its being roughly a language for “a mathematical theory 
of computation” 


_ 196l| based on “recursive functions of symbolic expressions” 

1959l| and surely “an algebraic language for the manipulation of symbolic ex- 


1958a I. 


pressions 

As McCarthy noted in the abstract, AIM-8 contained “only the machine[- 
]independent parts of the system”. We see for the first time lists be presented 
without mentioning memory addresses. In other words, in AIM-8, list got ab¬ 
stracted as a data type. Operations on lists were defined by axioms. The defi¬ 
nitions of the selector functions (called first and restP^ rather than car and 
cdr) and the constructor function (called combine instead of cons) 

I 959 I pp. 3-4], are reproduced below: 


McCarthy 


first [(e)] 

= e 

first [(ei,es)] 

= ei 

rest [(e)] 

= 0 

rest [(ei .e^)] 

= (ej 

combine [e;()] 

= (e) 

combine [ei ; (e^)] 

= (ei, 


where Cs might be “ 62 , ... ,e„” for n>2. McCarthy further noted that first 
and rest are defined only for S-expressions “which are neither null nor atomic”, 
and that combine is defined when e^, is not atomic. Note that the S-expression 
(), which represents a null list, was not considered an atomic symbol. The 
constraint on the second argument of combine prevents the construction of 
pairs, and in turn improper lists. 

Renaming the selector fun ctions shows that McCarthy “felt uneasy with th e 
machine[-]dependent names” [Stovanlll99lL p. 416] already in use since [1958a |. 


Constraining the the second argument of the constructor function to lists sug¬ 
gests that he probably recognized the possible misuse of the too-liberal con¬ 
structor function to build improper lists. 

The attempt to rename the selector functions failed and McCarthy reverted 
to the cryptic names car and cdr, as “the LISP community was already more 


powerful [than] the designer” |StovaDlll99lL p. 416]. The attempt to constrain 


the constructor function, was also abandoned, in our opinion, for no good reason 
as well. 


2.3 A symmetric variant 


At the end of AIM-8, McCarthy prop osed “binary Lisp” which was a variant 
that admits “only two[-]element lists” 19591 p. 17]. In particular, the status of 


^^These two names were reintroduced by the PLT people 
(http://racket-lang.org/people.html) into their variant of Scheme (now called Racket, 
http://racket-lcuig.org/) for car and cdr constrained (by their contract system) to 
accepting only proper lists as valid input. 
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first and rest was symmetric by definition: 


first [(ei,62)] = ei 
rest [(ei ,62)] = 62 
combine [ei; 62 ] = ( 61 , 62 ) 


What this definition would give is surely not “two-element lists” according to 
the definition given earlier, and had better be called pairs. McCarthy obviously 
abused the notation for lists here. 

Right below the presentation of this definition, McCarthy remarked, in this 
binary variant, that only two predicates = (symbol equality) and atom are needed 
(null dropped), and that “the null list can be dispensed with” [ibid, p. 17]. This 
should not be interpreted as that he tried to ditch the notio n of n ull list because 
a mathematician knows well the importance of null and in |l960l| he introduced 
NIL exactly for it. The correct interpretation is that he proposed eliminating 
() as a separate case in the definition of S-expressions, and treating it simply 
as an atomic symbol. McCarthy further pointed out that “the system is easier 
until we try to represent functions by expressions [...]” [ibid, p.l7]. Here, he 
probably meant that the system would lose its easy feel to the verbose nesting 
of pairs for building lists to represent F-e xpressions. 

The system given in McCarthy 1960j| turned out to be exactly this binary 
variant, albeit reverted to the function names car, cdr and cons. There Mc¬ 
Carthy presented the simplified definition of S-expressions [ibid, p. 187] as we 
know today: 


1 . Atomic symbols are S-expressions. 

2. If 61 and 62 are S-expressions, so is ( 61 . 62 ). 


He also resolved all the issues regarding binary Lisp as we see now and he saw 
then. The two components of a pair was separated by a dot instead of a comma. 
The atomic symbol NIL was chosen to mark the end of a list. The list notation 
( 61 , 62 , ... , 6 „) was defined as syntactic sugar for (ei. (02 . (... ( 6 „ . NIL) ...))) 

The system indeed feels simpler. However, it also exposes the underlying 
representation of lists. Naturally, the constraint on the constructor function of 
lists was abandoned so as to allow the construction of pairs, and the uncon¬ 
strained cons which mirrors the blind behavior of the corresponding machine 
instruction returned. As a consequence, improper lists found their way back. 
Every function expecting a list as argument now should test the argument to 
see if it is indeed proper, otherwise, it would fail when it receives instead an 
improper list or a circular list. But given that improper lists are seldom used 
and the test usually has a linear-time complexity. Lisp programmers either leave 
it out and assume the input to be a proper list by wishful thinking, or treat the 
last cdr as NIL in the case of an improper list and let the trap into an endless 
loop open in the case of a circular list. 
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2.4 F-expressions vs. S-expressions 


McCarthy him self al ways preferred and expected to “[write] programs as M- 
expressions”El [l978L p. 179]. Before the first implementation of Lisp came 
out, programs were indeed written as F-expressions and then hand-compiled 
to assembly code0 These two facts give us a good reason to believe that 
the implementation McCarthy expected was a compiler that compiles Lisp pro¬ 
grams directly or indirectly via S-expressions to assembly code. Our belief is 
also supported by McCarthy’s o wn words. His remark “you are confusing the¬ 
ory with practice” Stovanl[l984 p. 307] on Russell’s proposal of programming 
the universal functioio^ an assembly language by hancf^ suggests that, at 
that time he was not immediately aware that Russell had proposed an imple¬ 
mentation of Lisp by interpreting intermediate program representations in the 
form of S-expressions. Under Russell’s proposal, once source programs (in the 
form of F-expressions) are translated into intermediate programs (completely 
S-expressions) fo llowing the rules of translation first described in formally in 
McCartti^Il959j| and later specified formally in McCarthy 1960j| . the hand- 


compiled implementation of the universal function could readily interpret them. 
McCarthy later did realize that what Russell obtained b y hand-comp ilation of 
the universal function “certainly was” a Lisp interpreter Stovanlll984 p. 307]. 

What followed, which was probably one of the most dramatic events in 
the history of programming languages, that early adopters of Lisp went ahead 
programming in the intermediate language of S-expressions rather than the 
source language of F-expressions, was totally against McCarthy’s expectation! 
There might be technical reasons (for example, the translation scheme from F- 
expressions to S-expressions was not implemented yet) or historical factors (for 
instance, Russell advertised his result as an interpreter for Lisp source programs) 
for the incident. However, we believe the crucial reason is that the S-expression 
language (S-language), although used as intermediate language, was high-level 
enough for programming and even facilitating program construction. Indeed, 
the S-language is as high-level as the F-language (of F -expressions), since what 
gets changed through the “trivial” McCarthy Il959l p. 2] translation is only 
notation, not abstraction as in later-developed systems that translate source 
programs in some high-l evel language to some low- level intermediate language 
like the JVM b yte code Lmdhohna^ Yelli4 1999l | or the LLVM intermediate 


representation Lattner and Ad^ 


^^McCarthy’s expectation was fulfilled in the short-lived LISP 2 [Abrahams et alii 196611 . 

^^The assembly language was SAP (Symb olic Assembly Pro grams) for IBM 704. _ 

^^T he universal function, eval according to lMcCarthvI 119781 but apply according to lStovanl 

[20081 , was an F-expression. _ 

^^For other possible versions of the story, see [Stovanll2Q08|| . Whatever version, the story 
shows how difficult but also how important it is for theoreticians and practitioners to com¬ 
municate. 


7 







































3 Conclusion 


This investigation into the early development of Lisp shows how the look and 
feel of the language was shaped by mathematics and mechanics. The language 
was designed for writing programs algebraically. The algebraic feel was re¬ 
flected in functional expressions (or F-expressions). The intention to represent 
them, internally in the machine led to the introduction of the list data struc¬ 
ture, and externally in an intermediate language to the invention of symbolic 
expressions (or S-expressions). Gradually, list got abstracted as a data type 
and S-expressions admitted into the source-level language!^ After the first 
interpreter-based implementation of Lisp was running, S-expressions wan out 
as the preferred language for programming. In addition to technical and his¬ 
torical reasons, the incident could also be credited to the identical abstractive 
power of S-expressions with F-exp ressions. 

The design presented in |l959l | suggests that the very basic compound data 
type McCarthy wanted to include into Lisp was list, not pair. This suggestion 
was justified by McCarthy’s adherence to the mathematical notion of sequence 
in AIM-1 through AIM-8. After all, to process lists was one of the design 
goals of the language. Moreover, two-element lists cover all possible use cases 
of pairs. Some people may try to defend the status of pairs by appealing to 
space efficiency (since when storing two elements, a pair uses one less cons-cell 
than a list) or to obscure data structures (such as circular lists that represent 
infinitely-repeated sequences). However, now that space is no longer a big issue 
and the functionality of circular lists can be simulated by non-circular ones with 
a loop (more precisely, with a jump back to the start at the end), the existence 
of pairs in the language has become obsolete. 

Examining the origin of the binary variant, we sense a mathematician’s 
commitment to symmetry and reducti onism, which i s yet another “influence of 
the designer on the design” but which IStovan 1991 1 has probably overlooked. 
The rationale McCarthy explicitly gave for proposing the binary variant was that 
“the u nsymmetrical status of first and rest may be a source of uneasiness” 
I 959 L p. 17]. The one he implicitly held was naturally for simplifying the system 


by reducing lists to nested pairs. However, both rationales were weakened by 
the resultant system. The symmetry was never used. Instead, the asymmetry 
he tried to eliminate was reintroduced, not by constraint but by convention. 
The reduction led to the dilemma we have seen, where the programmer either 
does nothing sane to bear it or something insane to circumvent it. We believe 
a well-designed language should never put the programmer in such an awkward 
situation. One may propose including another set of manip ulation functions , 
say first, rest and combine as defined in the earlier part of McCarth^ll959l| . 
particularly for lists. However, it does not really solve the problem, only further 
complicates the system. If improper lists and circular lists are rarely used and 
can always be simulated in the rare case, it is better to kick them out to favor 
the common case. We urge designers of new Lisp dialects to discard pairs and 


“The rationale for this admission will be covered in the second essay of this series. 

















return to lists as presented in McCarth^ll959j |. 

This concludes the essay. In the following series, we will reveal other mys¬ 
terious aspects of Lisp such as the relationship between code and data, the 
existence of a meta-circular interpreter, etc. Lisp, as the first language that em¬ 
braces and integrates ideas from three major theoretical bases of computation, 
namely Turing machine, lambda calculus and recursion theory, is a goldmine 
worth deep digging. 
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